State-of-the-Art in Empirical Studies

Software engineering processes depend on the context they are applied in. Thus, it is risky to determine the best processes for a given project context without empirical data from this context. This report summarizes important issues on the state of the art in empirical studies in order to provide researchers with the background to conduct their own empirical studies and for practitioners to classify existing empirical studies regarding their usefulness. This report gives an overview on the state of the art in empirical studies in software engineering by prescribing a high-level process for empirical studies , which is refined for the three most often used empirical strategies: Controlled experiments, case studies, and surveys. For each strategy this work discusses representative application reports. The appendix lists a glossary of empirical vocabulary, a standard report outline , a bibliography, and experimental material available on-line. 3 Controlled Experiments 19 3.1 Study Definition 21 3.1.1 Determine the goal of the study and an informal list of hypotheses 22 3.1.2 Record the goal in the measurement goal template 23 3.1.3 Determine context of the experiment 25 3.2 Experiment design 26 3.2.1 Operationalize the goal by identifying and quantifying the dependent and independent variables 26 3.2.2 Select an experimental design that satisfies the experiment purpose 27 3.2.3 Consider validity threats 31 3.2.4 Determine the necessary criteria for potential subjects 32 3.2.5 Determine options for data collection to increase the potential benefit from the experiment 33 3.3